Skip to content

WIP: add image customization generation script #3358

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

maurerle
Copy link
Member

@maurerle maurerle commented Oct 27, 2024

this script creates the helper lua from the openwrt toh to be included in the image-customization.lua

closes #3316
supersedes #3143

Eventually, this should stay in community-packages or something less official, though I like that this properly depends on the wiki as well, making it easier to correct things for anyone.

I was made aware of the toh.json through this TOH viewer for Openwrt: https://github.com/soif/OpenWrtTOH/

@T-X
Copy link
Contributor

T-X commented Oct 28, 2024

Interesting. Do you have an example regarding how you use the image-customization-helper.lua in a site's image-customization.lua?

Could this toh.json also be used to simplify package/gluon-core/luasrc/usr/lib/lua/gluon/platform.lua?

If this were added to the main repository would refreshing of the image-customization-helper.lua also need to be added to the device integration checklist?

I'm also wondering if it were possible to avoid having to refresh and recommit the image-customization-helper.lua. (maybe just generate image-customization-helper.lua during a "make update" and don't add it to Git? Write it to ./gluon/tmp/ or ./gluon/output/? Or at least somewhere that's in a .gitignore. And make the result available to image-customization.lua that doesn't require explicitly including the helper file?)

@maurerle
Copy link
Member Author

Hi @T-X,
thanks for your comment! :)

Interesting. Do you have an example regarding how you use the image-customization-helper.lua in a site's image-customization.lua?

my current (but untested) commit to this is ffac/site@89d627d which makes the idea clear.

Could this toh.json also be used to simplify package/gluon-core/luasrc/usr/lib/lua/gluon/platform.lua?

as said by @neocturne the platform.lua uses a different format (for example requires to have ubnt instead of ubiquity and uses commas instead).

Furthermore, my current approach using the device id fails for some devices as well (for example for the unifi_outdoorplus and the ubiquiti-unifi-swiss-army-knife-ultra (which is weird, as there is no occurance of swiss in the toh.json). So we should make sure that all devices in the toh are matched by the correct identifier.

If this were added to the main repository would refreshing of the image-customization-helper.lua also need to be added to the device integration checklist?

yes, that would be the case (and making sure that the data is correct in the toh, which is cumbersome, as I heard that the openwrt wiki was separated from the development by intention, so I don't think it is helpful to tie these together here.
That's also why it is helpful to have this in community-packages instead, depending on general willingness of having this here.. :)

I'm also wondering if it were possible to avoid having to refresh and recommit the image-customization-helper.lua. (maybe just generate image-customization-helper.lua during a "make update" and don't add it to Git? Write it to ./gluon/tmp/ or ./gluon/output/? Or at least somewhere that's in a .gitignore. And make the result available to image-customization.lua that doesn't require explicitly including the helper file?)

automatic generation adds a side effect to the build process which might cause weird problems "package x does not include y anymore" - "oh someone edited the wiki" or "build failed, openwrt was down" and especially for building older releases it is required to have the matching helper.lua at hand, so I am quite sure this should be added to the VCS and updated manually.

@maurerle maurerle force-pushed the image_customization_utils branch from 21b0165 to a5f18e5 Compare January 19, 2025 19:55
@maurerle
Copy link
Member Author

maurerle commented Jul 3, 2025

The following 116 models do not have a matching device id in openwrt wiki:

Without matching Device ID
['gl-mt300a',
 'gl-mt300n',
 'gl-mt750',
 'nexx-wt3020-8m',
 'tp-link-archer-c2-v1',
 'tp-link-archer-c20-v1',
 'tp-link-archer-c20i',
 'xiaomi-miwifi-mini',
 'mikrotik-hap-ac2',
 'raspberrypi-2-model-b',
 'hewlett-packard-msm460',
 'x86-legacy',
 'netgear-nighthawk-x4s-r7800',
 'mikrotik-routerboard-951ui-2nd-hap',
 'mikrotik-routerboard-wapr-2nd',
 'sophos-red-15w-rev.1',
 'lemaker-banana-pi',
 'lamobo-r1',
 'avm-fritz-box-3370-rev-2-hynix-nand',
 'avm-fritz-box-3370-rev-2-micron-nand',
 'avm-fritz-box-7360-sl',
 'avm-fritz-box-7362-sl',
 'arcadyan-vgv7510kw22',
 'asus-rt-ac57u-v1',
 'cudy-x6-v1',
 'tp-link-archer-ax23-v1',
 'tp-link-eap615-wall-v1',
 'xiaomi-mi-router-4a-gigabit-edition',
 'xiaomi-mi-router-4a-gigabit-edition-v2',
 'xiaomi-mi-router-3g',
 'xiaomi-mi-router-3g-v2',
 'zbtlink-zbt-wg3526-16m',
 'zbtlink-zbt-wg3526-32m',
 'x86-64',
 'd-link-aquila-pro-ai-m30-a1',
 'gl.inet-gl-mt2500',
 'openwrt-one',
 'raspberrypi-3-model-b',
 'x86-generic',
 'linksys-ea6350v3',
 'meraki-mr33-access-point',
 'netgear-ex6100v2',
 'netgear-ex6150v2',
 'openmesh-a42',
 'openmesh-a62',
 'linksys-e8450-ubi',
 'ubiquiti-unifi-6-lr-v1',
 'cudy-wr1000',
 'gl-mt300n-v2',
 'tp-link-archer-c20-v4',
 'vocore2',
 'xiaomi-mi-router-4a-100m-edition',
 'xiaomi-mi-router-4a-100m-international-edition',
 'xiaomi-mi-router-4a-100m-international-edition-v2',
 'xiaomi-mi-ra75',
 'x86-geode',
 'tp-link-td-w8970',
 'linksys-e4200-v2-viper',
 'linksys-ea4500-viper',
 'buffalo-wzr-hp-ag300h',
 'buffalo-wzr-hp-g300nh-rtl8366s',
 'devolo-dlan-pro-1200+-wifi-ac',
 'd-link-dir825b1',
 'gl.inet-6416',
 'librerouter-v1',
 'netgear-wndr3700',
 'netgear-wndr3800ch',
 'netgear-wnr2200-8m',
 'netgear-wnr2200-16m',
 'openmesh-a40',
 'openmesh-a60',
 'openmesh-mr600-v1',
 'openmesh-mr600-v2',
 'openmesh-mr900-v1',
 'openmesh-mr900-v2',
 'openmesh-mr1750-v1',
 'openmesh-mr1750-v2',
 'openmesh-om2p-v1',
 'openmesh-om2p-v2',
 'openmesh-om2p-v4',
 'openmesh-om2p-hs-v1',
 'openmesh-om2p-hs-v2',
 'openmesh-om2p-hs-v3',
 'openmesh-om2p-hs-v4',
 'openmesh-om2p-lc',
 'openmesh-om5p',
 'openmesh-om5p-ac-v1',
 'openmesh-om5p-ac-v2',
 'openmesh-om5p-an',
 'sophos-ap100',
 'sophos-ap100c',
 'sophos-ap55',
 'sophos-ap55c',
 'teltonika-rut230-v1',
 'tp-link-archer-c5-v1',
 'tp-link-archer-c6-v2-eu-ru-jp',
 'tp-link-archer-c7-v2',
 'tp-link-tl-wdr4300-v1',
 'tp-link-tl-wr810n-v1',
 'tp-link-tl-wr2543n-nd',
 'ubiquiti-unifi-ac-lite',
 'ubiquiti-unifi-ac-lr',
 'ubiquiti-unifi-ac-pro',
 'ubiquiti-unifi-ap-outdoor+',
 'ubiquiti-unifi-swiss-army-knife-ultra',
 'friendlyelec-nanopi-r2s',
 'friendlyelec-nanopi-r4s',
 'raspberrypi-4-model-b',
 'raspberrypi-model-b',
 'armsr-armv8',
 'aerohive-hiveap-121',
 'gl.inet-gl-ar300m-nor',
 'gl.inet-gl-ar750s-nor',
 'netgear-wndr4300',
 'zyxel-nbg6716',
 'armsr-armv7']

The following 172 supported devices have a matching id:

With matching Device ID
['asus-rt-ac51u',
 'netgear-ex3700',
 'netgear-ex6130',
 'tp-link-archer-c50-v1',
 'mikrotik-sxtsq-5-ac-rbsxtsqg-5acd',
 'aerohive-hiveap-330',
 'enterasys-ws-ap3710i',
 'extreme-networks-ws-ap3825i',
 'ocedo-panda',
 'tp-link-archer-c2600',
 'ubiquiti-unifi-ac-hd',
 'linksys-wrt1200ac',
 'enterasys-ws-ap3715i',
 'tp-link-tl-wdr4900-v1',
 'lemaker-banana-pro',
 'avm-fritz-box-7360-v2',
 'avm-fritz-box-7412',
 'asus-rt-ax53u',
 'cudy-wr1300-v1',
 'cudy-wr2100',
 'cudy-x6-v2',
 'd-link-dap-1620-b1',
 'd-link-covr-x1860-a1',
 'd-link-dap-x1860-a1',
 'd-link-dir-860l-b1',
 'd-link-dir-878-a1',
 'd-link-dir-882-a1',
 'genexis-pulse-ex400',
 'gl.inet-gl-mt1300',
 'mercusys-mr70x-v1',
 'netgear-eax12',
 'netgear-ex6150',
 'netgear-r6220',
 'netgear-r6260',
 'netgear-wac104',
 'netgear-wax202',
 'netgear-wndr3700-v5',
 'tp-link-archer-c6-v3',
 'tp-link-re500-v1',
 'tp-link-re650-v1',
 'ubiquiti-unifi-6-lite',
 'ubiquiti-unifi-nanohd',
 'wavlink-ws-wn572hp3-4g',
 'zyxel-nwa50ax',
 'zyxel-nwa55axe',
 'zyxel-wsm20',
 'ubiquiti-edgerouter-x',
 'ubiquiti-edgerouter-x-sfp',
 'asus-rt-ax52',
 'asus-tuf-ax4200',
 'asus-tuf-ax6000',
 'd-link-aquila-pro-ai-m60-a1',
 'cudy-ap3000-outdoor-v1',
 'cudy-tr3000-v1',
 'cudy-wr3000-v1',
 'gl.inet-gl-mt3000',
 'netgear-wax220',
 'ubiquiti-unifi-6-plus',
 'wavlink-wl-wn573hx3',
 'zyxel-nwa50ax-pro',
 'aruba-ap-303',
 'aruba-ap-303h',
 'aruba-ap-365',
 'avm-fritz-box-4040',
 'avm-fritz-box-7530',
 'avm-fritz-repeater-1200',
 'gl.inet-gl-ap1300',
 'gl.inet-gl-b1300',
 'plasma-cloud-pa1200',
 'plasma-cloud-pa2200',
 'zte-mf289f',
 'zyxel-nbg6617',
 '8devices-jalapeno',
 'extreme-networks-ws-ap3915i',
 'cudy-tr1200-v1',
 'gl.inet-microuter-n300',
 'gl.inet-vixmini',
 'netgear-r6020',
 'netgear-r6120',
 'ravpower-rp-wd009',
 'tp-link-archer-c20-v5',
 'tp-link-archer-c50-v3',
 'tp-link-archer-c50-v4',
 'tp-link-re200-v2',
 'tp-link-re200-v3',
 'tp-link-re200-v4',
 'tp-link-tl-mr3020-v3',
 'tp-link-tl-mr3420-v5',
 'tp-link-tl-mr6400-v5',
 'tp-link-tl-wa801nd-v5',
 'tp-link-tl-wr841n-v13',
 'tp-link-tl-wr902ac-v3',
 'tp-link-tl-wr902ac-v4',
 'xiaomi-mi-router-4c',
 'avm-fritz-box-7312',
 'netgear-dgn3500b',
 'tp-link-td-w8980',
 'alfa-network-ap121f',
 'avm-fritz-box-4020',
 'avm-fritz-wlan-repeater-300e',
 'avm-fritz-wlan-repeater-450e',
 'avm-fritz-wlan-repeater-1750e',
 'buffalo-wzr-600dhp',
 'devolo-wifi-pro-1200e',
 'devolo-wifi-pro-1200i',
 'devolo-wifi-pro-1750c',
 'devolo-wifi-pro-1750e',
 'devolo-wifi-pro-1750i',
 'devolo-wifi-pro-1750x',
 'd-link-dap-1330-a1',
 'd-link-dap-1365-a1',
 'd-link-dap-2660-a1',
 'd-link-dap-2680-a1',
 'd-link-dap-2695-a1',
 'd-link-dir-505',
 'enterasys-ws-ap3705i',
 'extreme-networks-ws-ap3805i',
 'gl.inet-gl-ar150',
 'gl.inet-gl-ar300m-lite',
 'gl.inet-gl-ar750',
 'gl.inet-gl-usb150',
 'joy-it-jt-or750i',
 'netgear-wndr3700-v2',
 'netgear-wndr3800',
 'netgear-wndrmac-v2',
 'ocedo-koala',
 'ocedo-raccoon',
 'onion-omega',
 'plasma-cloud-pa300',
 'plasma-cloud-pa300e',
 'siemens-ws-ap3610',
 'sophos-ap15',
 'sophos-ap15c',
 'tp-link-archer-a7-v5',
 'tp-link-archer-c2-v3',
 'tp-link-archer-c25-v1',
 'tp-link-archer-c58-v1',
 'tp-link-archer-c7-v4',
 'tp-link-archer-c7-v5',
 'tp-link-archer-c59-v1',
 'tp-link-archer-c60-v1',
 'tp-link-archer-d50-v1',
 'tp-link-cpe210-v1',
 'tp-link-cpe210-v2',
 'tp-link-cpe210-v3',
 'tp-link-cpe220-v3',
 'tp-link-cpe510-v1',
 'tp-link-cpe510-v2',
 'tp-link-cpe510-v3',
 'tp-link-cpe710-v1',
 'tp-link-eap225-outdoor-v1',
 'tp-link-eap225-outdoor-v3',
 'tp-link-tl-wdr3500-v1',
 'tp-link-tl-wdr3600-v1',
 'tp-link-tl-wr842n-v3',
 'tp-link-tl-wr902ac-v1',
 'tp-link-tl-wr1043nd-v2',
 'tp-link-tl-wr1043nd-v3',
 'tp-link-tl-wr1043nd-v4',
 'tp-link-tl-wr1043n-v5',
 'tp-link-wbs210-v1',
 'tp-link-wbs210-v2',
 'tp-link-wbs510-v1',
 'ubiquiti-unifi-ac-mesh',
 'ubiquiti-unifi-ac-mesh-pro',
 'ubiquiti-unifi-ap',
 'ubiquiti-unifi-ap-pro',
 'friendlyelec-nanopi-r3s',
 'gl.inet-gl-e750',
 'gl.inet-gl-xe300',
 'netgear-wndr3700-v4',
 'zte-mf281']

I am currently mapping this using a custom device_mapping like this:

# mapping from openwrt device_id to gluon name
device_mappings = {
    "ubiquiti-unifi-uk-ultra": "ubiquiti-unifi-swiss-army-knife-ultra",
    "ubiquiti-unifi-outdoorplus": "ubiquiti-unifi-ap-outdoor+",
    "ubiquiti-unifi-ac-mesh-pro" : "",
    "ubiquiti-unifi-ac-mesh": "ubiquiti-unifi-ac-mesh-pro",
    "openwrt-one-1": "openwrt-one",
    "linksys-e8450": "linksys-e8450-ubi",
    "zyxel-nbg6716-a01": "zyxel-nbg6716",
    "cudy-x6": "cudy-x6-v1",
    "sophos-ap100-rev-1": "sophos-ap100",
    "sophos-ap100c-rev-1": "sophos-ap100c",
    "sophos-ap55-rev-1": "sophos-ap55",
    "sophos-ap55c-rev-1": "sophos-ap55c",
}

but this is of course one of the drawbacks of this approach.

Correctly receiving the toh.json automatically became much worse due to crawlers, as the service responds with 5xx if queried very soon again.

this script creates the helper lua from the openwrt toh to be included in the image-customization.lua
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

image-customization: add is_cellular_device and is_outdoor_device
2 participants